Real-time visual system for interaction with a humanoid robot

In this paper, we describe a new real-time visual system that enables a humanoid robot to learn from and interact with humans. The core of the visual system is a probabilistic tracker that uses shape and color information to find relevant objects in the scene. Multiscale representations, windowing and masking are employed to accelerate the data processing. The perception system is directly coupled with the motor control system of our humanoid robot DB. We present two case studies of on-line interaction with a humanoid robot: mimicking of human hand motion and smooth pursuit of human head motion. The generation of humanoid robot motion based on the position of relevant body parts is accomplished in real time. Both studies are supported by experimental results on DB. © 2001 Elsevier Science B.V. All rights reserved.